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DETAILED ACTION 

1 . This Office Action is in response to an Amendment filed by the Applicant on May 
15,2007. 

2. The Office Actions of February 6, 2007 and May 1 5, 2006 are incorporated into 
this Final Office Action by reference. 

Status of Claims 

3. Claims 1,2, 4-8, 10, 11, 13, 15, 18, 20 and 21 have been amended. Claims 16 
and 17 have been previously cancelled. Claims 1-15 and 18-21 are pending on this 
application. 

Information Disclosure Statement 

4. The information disclosure statement has not been filed for this application. To 
comply with 37 CFR 1 .98(a)(1), the following is required: (1) a list of all patents, 
publications, applications, or other information submitted for consideration by the Office; 
(2) U.S. patents and U.S. patent application publications listed in a section separately 
from citations of other documents; (3) the application number of the application in which 
the information disclosure statement is being submitted on each page of the list; (4) a 
column that provides a blank space next to each document to be considered, for the 
examiner's initials; and (5) a heading that clearly indicates that the list is an information 
disclosure statement. 

Claim Rejections - 35 USC §112 

5. In light of the amendments made to the claims, the rejection under 35 USC 112 
has been withdrawn. 
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Claim Rejections - 35 USC § 102 

* 

6. In light of the amendments made to the claims, the rejection under 35 USC 1 02 
has been withdrawn. 

Claim Rejections - 35 USC § 103 

7. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a 
person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived 
by the manner in which the invention was made. 

8. Claims 1-15 and 18-21 are rejected under 35 U.S.C. 103(a) as being 
unpatentable oyer Kevin Gurney in view of Hatayama ("An Introduction to Neural 
Networks", referred to as Gurney; US Patent #6,21 9,657; referred to as Hatayama). 

Claim 1 

Gurney anticipates an interconnecting neural network system (Gurney: page 1, 
L16-24) comprising: a neural network unit that includes a plurality of neurons (Gurney: 
page 1, L16-24; page 2, L19 to page 3, L9; Figs 1.2 and 1.3), each of the neurons 
outputting an excitation strength according to a similarity between input vectors and 
centroid vectors based on a kernel function (Gurney: page 1, L16-24; page 2, L19 to 
page 3, L9; page 14, L1-27; page 182, L15 to page 185, L21; Figs 1.2, 1.3, 2.4; 
Examiner's Note (EN): the activation (excitation strength) is calculated by the difference 
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||x-w|| (similarity). Radial Basis Function is a kernel function. Moreover, any function 
used by the neuron to provide its activation is a kernel function); and network control 
unit that constructs an artificial neural network structure by interconnecting neurons 
relating to each other among the neurons in the neural network unit via a weight 
(Gurney: page 1, L16-24; page 2, L19 to page 3, 19; Figs 1.2 and 1.3; EN: learning will 
develop a final structure for the neural network. The weights will relate the nodes in the 
final structure), wherein each of the neurons in the neural network unit outputs an • 
excitation strength according to the similarity between the input vectors and the centroid 
vectors based on the kernel function when each neuron is excited by the input vector 
applied from an outside (Gurney: page 1, L1 6-24; page 2, L19 to page 3, L9; page 14, 
L1-27; page 182, L15 to page 185, L21; Figs 1.2, 1.3, 2.4; EN: depending on the input 
vector, activation (excitation) will be calculated. The input vector must be provided from 
somewhere), and outputs a pseudo excitation strength obtained based on an excitation 
strength output from the other neuron when each neuron is excited in a chain reaction 
to excitation of the other neuron connected to each neuron (Gurney: page 1, L16-24; 
page 2, L19 to page 3, L9; page 14, L1-27; page 182, L15 to page 185, L21; Figs 1.2, 
1.3, 2.4; EN: the activation (excitation strength) of each neuron will be the input to a 
neuron in a successive layer (chain reaction). The activation calculated by the neurons 

« 

or the RBF units are considered pseudo excitation strength as understood from 
paragraph 60 of the present application), wherein the neurons in the neural network unit 
have a plurality of modalities different from one another (Gurney: pages 44-46, sections 
4.4 to 4.5.2; page 182, L15 to page 185, L21; EN: the association units will provide each 
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neuron the functionality of operating on different input data by dividing the input pattern 
into a grid and providing an output based on the input pattern, therefore providing 
different modalities for the neuron. Moreover, if the neuron can classify two classes, 
then it has a plurality of modalities, since it can provide a response for different input 
data. Moreover, each RBF will have a different centre and provide a contribution top 
the output). 

Gurney does not teach the plurality of modalities of the neurons including 
auditory modality and visual modality so that the neurons handle a plurality of different 
input vectors of auditory modality and visual modality. 

Hatayama teaches the plurality of modalities of the neurons including auditory 
modality and visual modality so that the neurons handle a plurality of different input 
vectors of auditory modality and visual modality (Hatayama: C3, L31 to C4, L2; C5, 
L15-22, C6, L 43-47; Figs. 1, 2, and 13; EN: the neurons receive camera information 
(visual modality) and voice information (auditory information). 

It would have been obvious to one of ordinary skill in the arts at the time of the 
applicant's invention to modify the teachings of Gurney by incorporating the plurality of 
modalities of the neurons including auditory modality and visual modality so that the 
neurons handle a plurality of different input vectors of auditory modality and visual 
modality as taught by Hatayama for the purpose of having a multifunction system • 
capable of operating on different types of data (Hatayama: C1 , L14-17). 
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Claim 2 

Gurney anticipates each neuron in the neural network unit outputs the pseuclo 
excitation strength and also outputs the centroid vector of each neuron when each 
neuron is excited in a chain reaction to the excitation of the other neuron connected to 
each neuron (Gurney: page 182, L15 to page 185, L21; EN: calculating the activation is 
calculating the activation is calculating the excitation strength. The centroid vector is the 
weight vector and it will be provided (outputted) to the next node connected (weighted 

i 

connections) to the node). 
Claims 3 and 9 

Gurney anticipates the network control unit interconnects the neurons relating to 
each other among the neurons in the neural network unit, based on an order of the 
neurons added or excited at time series in association with a plurality of input vectors 
applied to the neural network unit from the outside (Gurney: page 1, L16-24; page 2, 
L19 to page 3, L9; page 14, L1-27; Figs 1.2 and 1.3; EN: learning will develop a final 
structure for the neural network. During training, the neurons will be added (connected) 
or excited on a layer-by-layer basis. Since each neuron will have to wait to receive data 
from a neuron in a preceding layer, the interconnections will be made in time series). 
Claims 4 and 15 

Gurney anticipates the network control unit trains the weight that connects the 
neurons to each other, based on the excitation strength of each neuron in the neural 
network unit (Gurney: page 1, L16-24; page 2, L19 to page 3, L9; page 4, L10-31; page 
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t 

14, L1-27; pages 39-44, sections 4.1-4.4; EN: learning is training. Each neuron will 
provide an output as input to the nodes to which it is connected). 
Claims 5 and 10 

The network control unit removes each neuron at a predetermined timing 
determined based on the excitation strength of each neuron in the neural network unit 
(Gurney: page 1, L16-24; page 2, L19 to page 3, L9; page 4, L10-31; page 14, L1-27; 
pages 39-44,sections 4.1-4.4; EN: the weights will determine the strength of the 
connections, therefore connecting or disconnecting (removing) neurons during training). 
Claim 6 

Gurney anticipates each neuron in the neural network unit is an intermediate 
layer neuron using, as the centroid vector, centroid data in a matrix form in light of time 
series changes, and each intermediate layer neuron is connected to an output layer 
neuron that outputs a change in the excitation strength output from each intermediate 
layer neuron at time series (Gurney: page 1 82, L1 5 to page 1 85, L21 ; Fig1 0. 1 5; EN: a 
vector is considered a one row matrix. Training will update the weight vector (time 
series changes)). 
Claims 7, 11, 18 and 21 

Gurney teaches the kernel function employed in each neuron in the neural 
network unit includes a radial basis function (Gurney: page 182, L15 to page 185). 
Claim 8 

Gurney anticipates a method of constructing an interconnecting neural network 
structure (Gurney: page 1 , L16-24), the method comprising the steps of: preparing an 
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artificial neural network structure including a plurality of neurons (Gurney: page 1, L16- 
24; page 2, L19 to page 3, L9; Figs 1.2 and 1.3), each of the neurons outputting an 
excitation strength according to a similarity between an input vector and a centroid 
vector based on a kernel function (Gurney: page 1, L16-24; page 2, L19 to page 3, L9; 
page 14, L1-27; page 182, L15 to page 185, L21; Figs 1.2, 1.3, 2.4; Examiner's Note 
(EN): the activation (excitation strength) is calculated by the difference ||x-w|| (similarity). 
Radial Basis Function is a kernel function. Moreover, any function used by the neuron 
to provide its activation is a kernel function), the neurons relating to each other 
interconnected in the artificial neural network structure via a weight (Gurney: pagel, 
L16-24; page 2, L19 to page 3, L9; Figs 1.2 and 1.3; EN: learning will develop a final 
structure for the neural network. The weights will relate the nodes in the final structure); 
and training the weight that connects the neurons to each other, based on the excitation 
strength of each neuron (Gurney: page 1, L16-24; page 2, L19 to page 3, L9; page 4, 
L10-31; page 14, L1-27; pages 39-44, sections 4.1-4.4; learning is training. Each 
neuron will provide an output as input to the nodes to which it is connected), wherein 
the neurons in the artificial neural network structure have a plurality of modalities 
different from one another (Gurney: pages 44-46, sections 4.4 to 4.5.2; page 182, Ll5 
to page 185, L21; EN: the association units will provide each neuron the functionality of 
operating on different input data by dividing the input pattern into a grid and providing an 
output based on the input pattern, therefore providing different modalities for the 
neuron. Moreover, if the neuron can classify two classes, then it has a plurality of 
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modalities, since it can provide a response for different input data. Moreover, each RBF 

o 

will have a different centre and provide a contribution top the output). 

Gurney does not teach the plurality of modalities of the neurons including 
auditory modality and visual modality so that the neurons handle a plurality of different 
input vectors of auditory modality and visual modality. 

Hatayama teaches the plurality of modalities of the neurons including auditory 
modality and visual modality so that the neurons handle a plurality of different input 
vectors of auditory modality and visual modality (Hatayama: C3, L31 to C4, L2; C5, 
L15-22, C6, L 43-47; Figs. 1, 2, and 13; EN: the neurons receive camera information 
(visual modality) and voice information (auditory information). 

It would have been obvious to one of ordinary skill in the arts at the time of the 
« applicant's invention to modify the teachings of Gurney by incorporating the plurality of 
modalities of the neurons including auditory modality and visual modality so that the 
neurons handle a plurality of different input vectors of auditory modality and visual 
modality as taught by Hatayama for the purpose of having a multifunction system 
capable of operating on different types of data (Hatayama: C1, L14-17). 
Claims 12 and 19 

Gurney anticipates a computer readable recording medium storing an 
interconnecting neural network structure construction program that allows a computer to 
execute the method according to claim 8 (Gurney: page 1, L16-24; EN: Neural 
networks are computer systems. Computers must have programs stored in some ' 
medium in order to perform its operations). 
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Claim 13 

Gurney anticipates a method of constructing a self-organizing neural network 
structure including a plurality of neurons (Gurney: page 1 15 to page 118, section 8.2; 
page 185, L12-21), each of the neurons outputting an excitation strength according to a 
similarity between an input vector and a centroid vector based on a kernel function 
(Gurney: page 182, L15 to page 185, L21; Fig. 2.4; Examiner's Note (EN): the 
activation (excitation strength) is calculated by the difference ||x-w|| (similarity). Radial 
Basis Function is a kernel function), the neurons relating to each other being 
autonomously connected in the self-organizing neural network structure based on the 
input vector (Gurney: page 1 1 5 to page 1 22, section 8.2. 1 ), the method, comprising: a 
first step of adding a neuron, which has an input vector as a centroid vector for a kernel 
function, into the self-organizing neural network structure as a new neuron based on an 
input vector that is input first from an outside (Gurney: page 1 15 to page 122, section 
8.2.1; page 182, L15 to page 185, L21; EN: during training, lateral connections will be 
made based on the input vector); and a second step of repeating following processings 
(a) to (c), each of the processings being based on an input vector that is an n th input 
vector from the outside, where n is an integer equal to or greater than 2: (a) the 
processing of calculating excitation strengths of all the neurons in the self-organizing 
neural network structure based on the n th input vector input from the outside; (b) the 
processing of adding a neuron, which has the n th input vector as a centroid vector for a 
kernel function, into the self-organizing neural network structure as a new neuron in 
case that it is determined by the processing (a) that there is no neuron excited such that 
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the excitation strength thereof exceeds a predetermined threshold, among one or a 
plurality of neurons in the self-organizing neural network structure; and (c) the 
processing of performing both of or one of formation of a weight that connects the 
neurons, and training of the formed weight based on the excitation strengths of the 
neurons in the self-organizing neural network structure (Gurney: page 1 15 to page 120, 
L25; page 185, L1 2-21). 

Gurney does not teach the plurality of modalities of the neurons including' 
auditory modality and visual modality so that the neurons handle a plurality of different 
input vectors of auditory modality and visual modality. 

Hatayama teaches the plurality of modalities of the neurons including auditory 
modality and visual modality so that the neurons handle a plurality of different input 
vectors of auditory modality and visual modality (Hatayama: C3, L31 to C4, L2; C5, 
L15-22, C6, L 43-47; Figs. 1, 2, and 13; EN: the neurons receive camera information 
(visual modality) and voice information (auditory information). 

It would have been obvious to one of ordinary skill in the arts at the time of the 
applicant's invention to modify the teachings of Gurney by incorporating the plurality of 
modalities of the neurons including auditory modality and visual modality so that the 
neurons handle a plurality of different input vectors of auditory modality and visual 
modality as taught by Hatayama for the purpose of having a multifunction system 
capable of operating on different types of data (Hatayama: C1, L14-17). 
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Claim 14 

Gurney anticipates in the second step, a processing (d) of removing a neuron 
determined to be unnecessary based on the excitation strengths of the neurons in the 
self-organizing neural network structure is further performed (Gurney: page 116 to 
page118, section 8:1; page 127 to page 129, section 8.3.3; EN: inhibitory connections 
will remove neurons. Moreover, if a neuron is "off it has been removed). 

Claim 15 

Gurney anticipates each of the neurons in the self-organizing neural network 
structure holds a class label relating to a final output, and, in the processing (c) in the 
second step, only in case that the class label held by each neuron in the self-organizing 
neural network structure is identical, both of or one of the formation of the weight that 
connects the neurons, and the training of the formed weight is performed based on the 
excitation strengths of the neurons (Gurney: pages 135-136, section 8.3.6). 

Claim 20 

Gurney anticipates an interconnecting neural network system comprising: a 
plurality of intermediate layer neurons (Gurney: page 71, section 6.6, L1-4; page 72, 
Figs. 6.4 and 6.5; page 184, L13-16; Fig 10.15), each of the intermediate layer neurons 
outputting an excitation strength according to a similarity between an input vector and a 
centroid vector based on a kernel function (Gurney: page 1, L16-24; page 2, Li 9 to 
page 3, L9; page 14, L1-27; page 182, L15 to page 185, L21; Figs 1.2, 1.3, 2.4; 
Examiner's Note (EN): the activation (excitation strength) is calculated by the difference 
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||x-w|| (similarity). Radial Basis Function is a kernel function) and each of the 
intermediate layer neurons using centroid data in a matrix form in light of time series 
changes as the centroid vector (Gurney: page 182, L15 to page 185; Fig. 10.15; EN: a 
vector is considered a one row matrix. Training will update the weight vector (time 
series changes)); an output layer neuron connected to each of the intermediate layer 
neurons and outputting a change in the excitation strength output from each 
intermediate layer neuron at time series (Gurney: page 1, L16-24; page 2, L19 to page 
3, L9; page 14, L1-27; page 182, L15 to page 185, L21). 

Gurney does not teach the plurality of modalities of the neurons including 
auditory modality and visual modality so that the neurons handle a plurality of different 
input vectors of auditory modality and visual modality. 

Hatayama teaches the plurality of modalities of the neurons including auditory 
modality and visual modality so that the neurons handle a plurality of different input 
vectors of auditory modality and visual modality (Hatayama: C3, L31 to C4, L2; C5, 
L15-22, C6, L 43-47; Figs. 1, 2, and 13; EN: the neurons receive camera information 
(visual modality) and voice information (auditory information). 

It would have been obvious to one of ordinary skill in the arts at the time of the 
applicant's invention to modify the teachings of Gurney by incorporating the plurality of 
modalities of the neurons including auditory modality and visual modality so that the 
neurons handle a plurality of different input vectors of auditory modality and visual 
modality as taught by Hatayama for the purpose of having a multifunction system 
capable of operating on different types of data (Hatayama: C1, L14-17). 
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Response to Applicant's arguments 

9. The Applicant's arguments have been fully considered but are moot in view of 
new grounds of rejection necessitated by amendment. 

Examination Considerations 

1 0. The claims and only the claims form the metes and bounds of the invention. 
"Office personnel are to give the claims their broadest reasonable interpretation in light 
of the supporting disclosure. In re Morris, 127 F.3d 1048, 105455, 44USPQ2d 1023, 
1027-28 (Fed. Cir. 1997). Limitations appearing in the specification but not recited in the 
claim are not read into the claim. In re Prater, 415 F.2d, 1393, 1404-05, 162 USPQ 541, 
550-551 (CCPA 1969)" (MPEP p 2100-8, c 2, 1 45-48; p 2100-9, c 1, 1 1-4). The 
Examiner has full latitude to interpret each claim in the broadest reasonable sense.- 
Examiner will reference prior art using terminology familiar to one of ordinary skill in the 
art. Such an approach is broad in concept and can be either explicit or implicit in 
meaning. 

1 1 . Examiner's Notes are provided with the cited references to prior art to assist'the 
applicant to better understand the nature of the prior art, application of such prior art 
and, as appropriate, to further indicate other prior art that maybe applied in other office 
actions. Such comments are entirely consistent with the intent and spirit of compact 
prosecution. However, and unless otherwise stated, the Examiner's Notes are not prior 
art but a link to prior art that one of ordinary skill in the art would find inherently 
appropriate. 
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Conclusion 

12. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 

§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 

* 

mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

13. Claims 1-15 and 18-21 are rejected. 

Correspondence Information 

14. Any inquires concerning this communication or earlier communications from the 
examiner should be directed to Omar F. Fernandez Rivas, who may be reached 
Monday through Friday, between 8:00 a.m. and 5:00 p.m. EST. or via telephone at 
(571 ) 272-2589 or email omar.fernandez rivas@uspto.gov. 

If you need to send an Official facsimile transmission, please send it to (571) 
273-8300. 
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If attempts to reach the examiner are unsuccessful the Examiner's Supervisor, 
David Vincent, may be reached at (571 ) 272-3080. 

Hand-delivered responses should be delivered to the Receptionist @ (Customer 
Service Window Randolph Building 401 Dulany Street Alexandria, VA 22313), located 
on the first floor of the south side of the Randolph Building. 



Omar F. Fernandez Rivas 
Patent Examiner 
Artificial Intelligence Art Unit 2129 
United States Department of Commerce 
Patent & Trademark Office 



Monday July 23, 2007 





